Abstract
Background: Decision-making under deep uncertainty requires robust methodological approaches that can handle incomplete information and multiple stakeholder perspectives. This review systematically maps the methodological landscape across 15,000+ publications.
Methods: We employ automated text mining, keyword classification, and bibliometric analysis to categorize methods, applications, and trends in decision support under uncertainty.
Keywords: decision support, deep uncertainty, value of information, Bayesian analysis, stochastic optimization, multi-criteria decision analysis, systematic review, bibliometrics
Decisions made under uncertainty occur in circumstances where a decision-maker is in the position of choosing between potentially competing options. Such decisions often have to be made with no formal scientific support and often do not need much formal support (Keeney, 2004). Decision-supporting research can offer formal support by prescribing a recommended course of action. In general, this is done with the assumption that decision-makers seek maximum expected utility. This means we must assume that the decision maker will want the option with the greatest positive outcome according to their stated outcomes of interest. The decision-making processes can be categorized into two levels of decision making: 1) a decision aimed at influencing an outcome where the expected choice is the one which maximizes the desired outcome, based on information currently available to the decision maker and 2) a decision on how to allocate resources to reduce the uncertainty in the previous decision problem, i.e., to increase the current information to improve the likelihood of choosing the option with the most beneficial outcome.
Such decisions are often made in policy settings like in healthcare, where a choice to fund the health of one part of society often comes at the cost of another, assuming limited budget allocation (Claxton et al., 2015). This choice involves costs, benefits and risks in the form of possible damage and trade-offs in comparison to a baseline and or other decision options.
Our own work has covered such situations regarding the nutritional costs of agricultural expansion in Uganda (C. W. Whitney et al., 2018), the choice between different siltation management options in Burkina Faso (Lanzanova et al., 2019), the management of competing water resources in Kenya (Luedeling et al., 2015), and deciding between agroforestry interventions in Vietnam (Do et al., 2020). In these cases, we apply, adopt and update the classical processes of Decision Analysis (Howard & Abbas, 2015; Jeffrey, 1990; Keeney, 1982). To achieve this, we co-generate impact pathways of decisions to provide the structure for a mathematical formula that is then used to estimate the expected utility. We perform a quantitative analysis (Luedeling et al., 2022) of welfare-based decision-making processes using Monte Carlo simulations. In our work, we interpret the welfare-based decision problem as a von Neumann-Morgenstern utility function and apply the ‘Value of Information Analysis’ to assign a value to a certain reduction in uncertainty or, equivalently, increase in information.
There are a number of ways that decisions made under uncertainty can be supported. Through this review we seek to generate a complete overview of all the applications of similar processes that aim to support decision-making under uncertainty.
We offer an overview of methods that are applied to support decisions under uncertainty. We focus on summarizing the state-of-the-art methods in decisions. We identify, define, and evaluate research by extracting the decision-supporting methods, describing them and their approach to supporting decisions. We present a protocol-driven comprehensive review and synthesis of the various methods applied for decision support. We offer an overview of where these methods are currently applied and some interpretation about where these applications can be expanded.
Protocol: We follow the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) (Figure 2). We gather data from the Web of Science Core Collection curated database of published, peer-reviewed and larger and more diverse published articles, preprints, theses, books, and other relevant scientific content in Google Scholar. After much deliberation and discussion amongst the authors, the final search query consisted of the following keywords.
Decision + Intervention OR policy + Uncertainty + Expert OR stakeholder + Model OR monte carlo OR simulation OR computer assisted + value of information OR information accuracy
“decision”+(“intervention”OR”policy”)+“uncertainty”+(“expert”OR”stakeholder”)+(“model”OR”monte carlo”OR”simulation”OR”Bayesian”OR”computer assisted”)+ (“value of information”OR”information accuracy”)
From Web of Science, 14 records were collected in May 2023, and from Google Scholar, 17500+ records were collected in April and May 2023. Associated papers and articles from the search records were also gathered as secondary data collection. Some had shared arXiv or doi links, other parts of shared contributions and assorted affiliations that led to new papers. We applied specific inclusion criteria to this large set of records to reduce the number of records covered under this study. An independent screening of the collected records is performed to confirm their eligibility for the study.
We use Keeney’s representation of where formal decision support is required and applied as motivation and framework for the assessment - Keeney’s judgment about how 10,000 decisions are typically made - his personal histogram of 10,000 decisions being faced by numerous decision makers @ref(#fig:01_keeney) (Keeney, 2004).
We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) @ref(fig:02_prisma) SystematicReviewOverviewFigure
We preserved our search string on searchRxiv (C. Whitney, 2023)
Google Scholar (GS) yielded 17,600 results but without an asterisk for decision* and model* it yielded just 17,400 results. This is a bit counter intuitive; the ’*’ should widen rather than narrow the search. If anything, we should expect the opposite effect and the larger return in the former search. Then these 200 additional results would say something like ‘models’ or ‘decisions’ but not ‘model’ or ‘decision’. These should be kept.
‘Google Scholar advanced search’ is much less targeted (2.4 million hits): with all of the words: decision intervention policy uncertainty model with at least one of the words: “value of information”, “information accuracy”, “intervention”, “policy”, “model*“,”monte carlo” “simulation”, “Bayesian”, “computer assisted”, “expert”, “stakeholder”, where my words occur ‘anywhere in the article’.
After importing papers to Zotero we:
In looking for the PDFs and following up with some of the thesis, notes, syllabi etc. in the Google Scholar returned list we added additional associated papers and articles. Some had shared arXiv or doi links, other parts of shared contributions and assorted affiliations that lead to new papers. Sometimes the class notes and syllabus that were in the scholar search return were there because they listed papers that were relevant to our search terms. These were sometimes already in our main collection and sometimes not. When they were not in our collection, we added them.
Web of Science - 14 results
CABI - 13 results
Decision + Intervention OR policy + Uncertainty + Expert OR stakeholder + Model OR “monte carlo” OR simulation OR “computer assisted” + “value of information” OR “information accuracy”
# Read the bib file
bib_data <- bib2df("bib/23_Methods_Review_Holistic_Systems.bib")
# Basic cleaning and text preparation
clean_bib_data <- bib_data %>%
mutate(
has_annotation = !is.na(ANNOTE) & ANNOTE != "",
annotation_text = ifelse(has_annotation, ANNOTE, ""),
full_text = paste(TITLE, ANNOTE, sep = " "),
year = as.numeric(YEAR)
)# method categories
method_categories <- list(
bayesian = c("bayesian", "bayes", "mcmc", "markov chain", "prior", "posterior"),
simulation = c("simulation", "monte carlo", "stochastic", "agent-based", "discrete event"),
optimization = c("optimization", "linear programming", "nonlinear programming", "heuristic"),
statistical = c("regression", "anova", "time series", "survival analysis", "mixed model"),
decision_analysis = c("decision analysis", "decision tree", "markov model", "value of information", "voi"),
machine_learning = c("machine learning", "neural network", "random forest", "svm", "clustering"),
multi_criteria = c("multi-criteria", "multi criteria", "analytic hierarchy", "ahp", "topsis"),
economic_evaluation = c("cost-effectiveness", "cost-benefit", "cost-utility", "economic evaluation"),
risk_analysis = c("risk analysis", "risk assessment", "sensitivity analysis", "uncertainty analysis")
)source("R/detect_methods_enhanced.R")
# Apply method detection
clean_bib_data <- clean_bib_data %>%
mutate(
detected_methods = map_chr(full_text, detect_methods_enhanced),
# Also detect from title only for confidence scoring
methods_from_title = map_chr(TITLE, detect_methods_enhanced)
)source("R/add_confidence_scores.R")
# Apply confidence scoring
clean_bib_data <- add_confidence_scores(clean_bib_data)## === METHOD DETECTION SUMMARY ===
## Total papers: 15200
## Papers with methods detected: 1828
## Papers with VOI: 263
## Confidence distribution:
##
## high medium none
## 1823 5 13372
##
## === SAMPLE OF DETECTED METHODS ===
sample_results <- clean_bib_data %>%
filter(detected_methods != "") %>%
select(TITLE, detected_methods, method_confidence) %>%
head(10)
print(sample_results)## # A tibble: 10 × 3
## TITLE detected_methods method_confidence
## <chr> <chr> <chr>
## 1 A hierarchical bayesian approach for inco… bayesian high
## 2 Integrating monitoring and optimization m… optimization high
## 3 A systematic review and economic evaluati… economic_evalua… high
## 4 The rocky road to extended simulation fra… simulation; opt… high
## 5 A practical guide to value of information… decision_analys… high
## 6 Late pregnancy ultrasound to screen for a… decision_analys… high
## 7 Efficient research design: {Using} value-… economic_evalua… high
## 8 Economic evaluation of transperineal vers… economic_evalua… high
## 9 The cost-effectiveness of a novel {SIAsco… economic_evalua… high
## 10 Bayesian design and analysis of external … bayesian high
source("R/analyze_method_frequency.R")
method_results <- analyze_method_frequency(clean_bib_data)
print(method_results$plot)(#fig:method_results_plot)TRUE
## # A tibble: 9 × 2
## detected_methods n
## <chr> <int>
## 1 economic_evaluation 433
## 2 decision_analysis 396
## 3 bayesian 380
## 4 simulation 238
## 5 risk_analysis 209
## 6 optimization 169
## 7 multi_criteria 77
## 8 machine_learning 64
## 9 statistical 22
source("R/analyze_voi_papers.R")
# Analyze VOI papers
voi_results <- analyze_voi_papers(clean_bib_data)## === VOI PAPERS ANALYSIS ===
## Number of VOI papers: 263
##
## Methods used in VOI papers:
## # A tibble: 6 × 2
## detected_methods n
## <chr> <int>
## 1 decision_analysis 263
## 2 economic_evaluation 20
## 3 bayesian 5
## 4 simulation 4
## 5 optimization 2
## 6 risk_analysis 2
(#fig:voi_results)TRUE
# Create a comprehensive results table
results_table <- clean_bib_data %>%
select(
TITLE,
YEAR = year,
HAS_ANNOTATION = has_annotation,
DETECTED_METHODS = detected_methods,
CONFIDENCE = method_confidence,
VOI = has_voi,
BAYESIAN = has_bayesian,
SIMULATION = has_simulation,
UNCERTAINTY = has_uncertainty
)
# Export for manual verification
write.csv(results_table, "data/method_detection_results.csv", row.names = FALSE)##
## === EXPORT SUMMARY ===
## Results exported to: method_detection_results.csv
## Total papers processed: 15200
## Papers with methods: 1828
## High confidence classifications: 1823
# Show top methods
top_methods <- clean_bib_data %>%
filter(detected_methods != "") %>%
separate_rows(detected_methods, sep = "; ") %>%
count(detected_methods, sort = TRUE) %>%
head(10)
cat("\nTop 10 methods detected:\n")##
## Top 10 methods detected:
## # A tibble: 9 × 2
## detected_methods n
## <chr> <int>
## 1 economic_evaluation 433
## 2 decision_analysis 396
## 3 bayesian 380
## 4 simulation 238
## 5 risk_analysis 209
## 6 optimization 169
## 7 multi_criteria 77
## 8 machine_learning 64
## 9 statistical 22
# Quick validation - check if our detection makes sense
validation_check <- clean_bib_data %>%
filter(has_voi) %>%
select(TITLE, detected_methods, method_confidence) %>%
arrange(desc(method_confidence))
cat("\n=== VALIDATION CHECK - VOI PAPERS ===\n")##
## === VALIDATION CHECK - VOI PAPERS ===
## # A tibble: 263 × 3
## TITLE detected_methods method_confidence
## <chr> <chr> <chr>
## 1 A practical guide to value of information… decision_analys… high
## 2 Late pregnancy ultrasound to screen for a… decision_analys… high
## 3 A cost-effectiveness and value of informa… decision_analys… high
## 4 Expected value of information analysis to… decision_analys… high
## 5 Are head-to-head trials of biologics need… decision_analys… high
## 6 Efficient value of information computation decision_analys… high
## 7 Supplier quality improvement: {The} value… decision_analys… high
## 8 Use of value of information in {UK} healt… decision_analys… high
## 9 Decisions on further research for predict… decision_analys… high
## 10 Towards social learning in water related … decision_analys… high
## # ℹ 253 more rows
# Check papers with high confidence
high_confidence <- clean_bib_data %>%
filter(method_confidence == "high") %>%
select(TITLE, detected_methods, methods_from_title)
cat("\n=== HIGH CONFIDENCE CLASSIFICATIONS ===\n")##
## === HIGH CONFIDENCE CLASSIFICATIONS ===
## # A tibble: 1,823 × 3
## TITLE detected_methods methods_from_title
## <chr> <chr> <chr>
## 1 A hierarchical bayesian approach for inc… bayesian bayesian
## 2 Integrating monitoring and optimization … optimization optimization
## 3 A systematic review and economic evaluat… economic_evalua… economic_evaluati…
## 4 The rocky road to extended simulation fr… simulation; opt… simulation; optim…
## 5 A practical guide to value of informatio… decision_analys… decision_analysis
## 6 Late pregnancy ultrasound to screen for … decision_analys… decision_analysis…
## 7 Efficient research design: {Using} value… economic_evalua… economic_evaluati…
## 8 Economic evaluation of transperineal ver… economic_evalua… economic_evaluati…
## 9 The cost-effectiveness of a novel {SIAsc… economic_evalua… economic_evaluati…
## 10 Bayesian design and analysis of external… bayesian bayesian
## # ℹ 1,813 more rows
# Minimal interactive table
DT::datatable(
clean_bib_data %>%
select(
Author = AUTHOR,
Year = YEAR,
Title = TITLE
) %>%
arrange(desc(Year), Author),
options = list(
pageLength = 5,
lengthMenu = c(5, 10, 20),
dom = 'tip' # table, info, pagination only
),
caption = "Reference List",
rownames = FALSE,
filter = "none" # remove filter to save space
)(#fig:datatable_clean_bib_data)TRUE